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IDENTIFIERS 
ABSTRACT 

Mail, telephone, and in-group methods of surveying 
youth are analyzed and an example of the use of the latter method 
during the early phase of Project 4-Health (4-H) in California is 
provided. Issues in data collection are associated with cost, time, 
survey item comprehension, and participation rates. Project 4-Haalth 
is a S-year tobacco prevention research project conducted by the 
University of California at Berkeley School of Public Health and the 
California 4-H program, under funding by the National Cancer 
Institute. The primary aim of the project is to develop and field 
test a smokeless tobacco prevention program to be delivered to youth 
in 4-H clubs throughout the state. In-group surveys of nearly 2,300 
4-H members were conducted in the fall of 1986 to assess tobacco use 
by the target population and social and psychological correlates of 
tobacco use. Administrators' reports, observations by researchers, 
and discussions with program participants indicated that: (1) youth 
experienced moderate problems in reading and comprehending 
instructions; (2) one-day training for survey administrators proved 
to be satisfactory; and (3) it was difficult to ensure privacy. 
(TJH) 
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Introduction 

The formats in which data are collected represent an issue 
of continuing interest for t^esearchers and evaluators. The 
overriding concern is for the quality of the data, but logistical 
issues relating to the timing, cost, and effort of data 
collection are important as well. This oeoer considers the 
relative merits of three modes of collecting survey data from 
young respondents: in-grouo, telephone, and mail. It also 
provides data pertaining to the success of group administration 
in an unusual setting. 

Some previous research has appeared comparing the telephone 
format with either personal interviews or mail surve/s (e.g.. 
Groves & Kahn, 1979; Sudman & Bradburn. 19741. Most of these 
studies have found that none has a clear advantage with regard to 
data Quality; decisions should be based on factors of staffing, 
available time, logistics, and cost. Hochstim (1967) compared 
telephone surveys with both mail surveys and personal interviews, 
and found that all three methods produced highly comparable data. 
Recent developments in computer-assisted telephone interviewing 
fCATT) have furthered enhanced the confidence that can be Placed 
in the telephone format. Beckman et al. fl987), however, 
compared telephone and mail responses for sensitive items and 
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found that the telephone format produced significantly fewer 
extreme responses (i.e., "l"s or "5"s on a Likert scale). 

Most of these studies have involved adult respondents; 
little is known about how these methods may compare when youth 
are the population of interest. Further, little has appeared on 
the option of admini st er incj surveys in a full-group setting. 
This seems due to the fact thst for the most part, such groupings 
can be found only for children and adolescents fi.e., 
classrooms). In studies involving these populations this option 
has routinely been used, while for studies involving adults this 
option has not been available. The health education research 
Project that led to this paper was an exception, in that all 
three options were both available and carefully considered. 
Issues in data collection 

The decision of which data collection format to use is a 
complex one. relying on a large number of factors. For surveys 
involving sensitive tcpics such as personal lifestyle, the 
complexity is compounded. Some of the important factors are the 
f ol lowing : 

o Cost of data c ollection . The priority of this topic in 
the decision-making process will vary across projects 
end. of course, depend on the extent of available 
resources. Cost might not be a critical concerr. if 
adeauate outside funding has been obtained for a study, 
but for many projects the need to control costs is an 
important consideration when deciding among 
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methodol ogical oDt ions. 

Time needed for data col lection , This can range from 
as little as one day (for a group administration at 
which all respondents are in attendance) to several 
weeks or months for telephone or mail surveys. Mail 
surveys in particular require a long minimum period of 
t ime, since three or more mai 1 ings must usual ly be 
Planned, with two or more weeks intervening between 
each. For methods requiring extended periods of data 
collection fe.g,, two or three months!, some concern 
about data bias may be warranted due to the possibility 
of historical events occurring in the interval, 
limiting the comparability of early- and late-arriving 
data. This may be particularly pertinent for surveys 
dealing with subjects that attract wide media coverage. 

Uhen allocating time for data collection, it is 
also wise to consider the need for data preparation. 
Considerable cleaning of data, including elimination of 
uninterpretable cases, may be involved when dealing 
with young participants. 

Training of data collectors . Mail surveys avoid the 
need to train survey administrators, but at the cost of 
surrendering much of the potential control over the 
data collection process. Telephone surveys usual ly 
involve trair-'ng of interviewers, unless the project is 
contracted to another organization (which can be done 
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only if the budget allows). Group administrations vary 
depending on the case: for most studies administration 
can probably be completed by the researchers 
themselves: when this is not possible (as in the 
present study) the training and expertise of survey 
administrators is a critical concern. Training car. 
present logistical difficulties when administrators are 
located across broad geographical areas, as was the 
case in the study described later in this paper. 
Comprehensibi I ity of items for respondents. This is a 
particularly important consideration when dealing with 
children and adolescents. It presents most concern 
when reading is involved rather than oral interview. 
Careful Piloting of items for readability can reduce 
the problem, but differences in reading ability among 
youth will make full comprehensibi 1 ii:y very difficult 
to achieve. Mail surveys are weakest in this area, 
since respondents must work in isolation. in group 
administrations, a researcher is available to remedy 
difficulties and answer Questions. Telephone 
interviews are probably strongest here, since reading 
is not involved. However, this does not entirely 
resolve the issue of comprehensibi 1 ity , and pilot 
testing is still an essential preliminary activity, 
even in telephone interviews, to ensure that items are 
clear 1/ understood. 
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o Maximizing participation rates from the tg^q« >ted 

sample . Mai 1 surveys present a major problem in this 
area, since very hi^jh response rates are rare, and 
survey researchers often plan in terms of a target 
response rate of 60% - 80%. (These estimates assume 
the use of multiple mailings.) Some approaches, 
notably Dillman's (1978), provide the promise for rates 
over 90%. but most surveys fall far short of this goal. 
In-group administrations usually involve a cercentage 
of the sample being absent for the data collection, and 
these individuals must be made up in a future visit (at 
which some absen^^eeism might still exist), or through 
another format, such as phone calls. Telephone surveys 
are relatively strong for achieving high participation, 
though provision must be made for the cost and time 
involved in multiple callback attempts. In all cases. 
100% participation is a very difficult target to reach. 
Wi th the mai 1 and in-group opt ions, reaching non- 
respondents and absentees frequently involves a change 
in formats (i.e., from mail to telephone calls), »vhich 
raises concern for data comparability. 

o S uscept ibi 1 ity to demand characteristics . For 

sensitive data, e.g., regarding the respondent's 
substance use history or attituaes toward illegal 
drugs, data validity will depend partly on the honesty 
and frankness with which the responses are given. Any 
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setting which increases pressure on the respondent to 
provide the self -perceived "socially desirable" 
response may potentially compromise the data. Formats 
that maximize the psychological sense of trust and 
privacy will be superior in encouraging honesty. Thus 
mail auest ionnaires, which can be filled out completel 
in private, probably have an advantage (assuming that 
parents or other household members grant the necessary 
orivacy). Group adrriinistrations have the potential 
Problem of a social setting, with both administrators 
and other respondents present. Telephone interviews 
may provide an eaual or greater potential concern, 
since they necessitate an individual interaction; 
whether this effect is tern^ered by the impersonal 
electronic connection is an important question for 
further investigation. It is notewortny that the 
isolation of the response setting afforded by the mail 
Questionnaire is double-edged: it presents an 
advantage with regard to the criterion of respondent 
honesty but a disadvantage with regard to respondent 
support (i.e., assisting with the comprehensibi 1 i ty of 
items) and maximizing the participation rate. 

Another aspect to the issue of minimizing social 
desirabilitv :-esponding is that the in-group setting 
can provide the opportunity to present secondary forms 
of support, persuasion* or data validation. For 
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ftxamole, in sMoking prevention research th/* biochemical 
validation of self-reoorts (Pechacek et al.^ 1984.) — 
involving the collection of saliva samples from 
respondents — is possible only in the group setting 
(barring costly added procedures such as supplemental 
home visits). The importance of retaining options for 
validating or supplementing the respondents' data 
depends on the goals, perspectives, and aims of the 
individual study. 
Table 1 presents a summary of these data collection 

concerns, providing a corriparison of the thre^ formats for each 

issue. 

A research example 

These- concerns were very relevant to recent data collection 
decision.^ made in Project d-Health, a tobacco prevention research 
project being conducted by the UC Berkeley School of Public 
health and the California 4.-H program. Several phases are 
involved in the project. The in-group format was used for a 
survey in an early phase. Information from that experience, plus 
our consideration of the factors listed above, informed our 
decisions regarding data collection in later phases. What we 
present in this report is not an experimental comparison of 
different formats, but the progress of our analysis and decision- 
makin9 in the course of our project . 

Project description- Project 4-Health is an ongoing five-- 
year health education project funded by the National Cancer 
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Institute. The primary aim is to develop and field test a 
smokeless tobacco prevention program, to be delivered to youth in 

clubs throughout California. The curriculum that is being 
developed actually provides a comprehensive tobacco focus, 
jointly addressing both cigarettes and smokeless tobacco. The 
field test of the program is scheduled to begin in early 198S. 

Several of the major preliminary research tasks were to 
assess the extent o-^^ use of all forms of tobacco in the target 
population, and to determine the Psychological and social 
correlates of such use. Accordingly, a survey of members was 

conducted in fall 19S6 to measure critical variables related to 
these questions. The decision about how to collect the survey 
data incorporated all of the concerns listed above. 

Research POPLlation and setting . In California. clubs 
meet once a month during the school year. Meetings are usually 
held on weekday evenings, in local schools or community centers. 
They are attended by members and club leaders, and usually 

several parents as well. The sizes of the clubs in our study 
ranged from 19 to ld6, with the median club size being LI 
members. Club members range in age from 9 to 19, with most being 
between 9 and 13 years old. 

The leaders of clubs ere adult volunteers, usually parents 
of current or former members. Older teen members are often used 
in leadership positions as well. Clubs within a county ar*e 
coordinated at the county level by professional staff members 
known as ^-H Advisors, who hold academic appointments in 
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University of California Cooperative Extension. 

The sample that was selected to participate in the fall 1986 
survey included 77 clubs in 25 counties, with approximately 2.500 
youth involved- The requested inf ormat ion included use of 
cigarettes and smokeless tobacco, beliefs and attitudes towarc 
tobacco and tobacco users, and background demographic 
information. 

In Planning the survey, we chose to test the in--club setting 
for its feasibility as a data collection site. tJe had severai 
methodological concerns about the suitJibility of group settings 
outside of schools, which could be resolved only through 
empirical trial* These concerns, which involved factors 
distinguishing the 4-H club setting from classroorri settings, 
included: (i) the wide range of ages among the youth; (ii) the 
variability in physical sites, in terms of placement of chairs, 
availability of tables or writing surfaces, overall space, and 
facilities to keep participants Physically separated from non- 
participants; (iii) the presence of parents an-i other adults; 
(iv) the prospects of absenteeism: (v) the potential to create 
auiet and studious conditions for participants. 

The survey was designed for anonymous response, and reauired 
about 25-35 minutes for youth to complete. It covered 16 pages 
and about 110 items. Virtually all items entailed selection from 
an array of responses rather than generation of information. 
Attitudinal items were presented in Likert format on a 5-Point 
scale. The Advisors who served as administrators were trained in 
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a full-day session in October 1986, ''n addition, a short 
eoucationol presentation on surveys was developed for 
presentation to the partxcipants immediately before distribution 
of the survey, to explain selected concepts end to build 
motivation for respondin^i honestly, 

'Measures and potential for feedback . In light of the lack 
of precedents for this form of data collection, we planned and 
carried out a careful monitoring scheme to provide information on 
the success of this approach. The monitoring plan clearly did 
not have the power of a controlled study comparing two or more 
formats, but given our priorities and resources we felt it could 
Provide a sufficient amount of information to guide our decisions 
on data collection for future phases. 

The Plan included asking each survey administrator to 
complete a repor.. outline noting the details of each club 
administration. The form included facts such as dates, time and 
attendance, information about on-task nd off-task behavior, and 
judgments about clarity of instructions, reactions of aiults 
present, level of interest sustained, and other items. In 
addition, administrators were asked to document whether each of 
the following were a Problem during administration, and if so, to 
estimate how many participants were adversely affected: 
difficulty in reading the survey; confusion about Question 
format; loss of interest in completing the survey; not taking the 
survey seriously; talking; distractions by survey participants; 
and distractions by survey non-participants. 
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In addition to the survey administrator's report, we 
followed a careful data screening procedure. A minimum number of 
pages completed was established for inclusion in the final Jata 
set. Furthermore, booklets that revealed an obvious pattern of 
careless responding were discarded. Altogether, 150 of the 
survey booklets were discarded as a result of this Procedure, 
resulting in a data set of 229S cases. 

The f i nal data set proved to be very comparable with other 
reports of tobacco use among youth, providing support for the 
reliability of the data. For example, 23% of our sample reported 
any lifetime experience with smokeless tobacco, and 4.6% reported 
use in the last 7 days. These figures are well within the range 
of estimates produced by local surveys, as noted in the recent 
Surgeon General's report (U.S. Dept. of Health and Human 
Services, 1986). 

P?esults; Assessment cf" t^^e jn-group ad^inj street i on pr^or^edur^e 

The administrators' reports, combined with our own 
observations and discussion with program participants, led to the 
following conicusions about the in-group administration in our 
project : 

1 „ Djf -Pjculties in reading and comprehension o-^ 1 nst r^uct 1 on^ 
were a nnoderate problem in the survey adminstrat ion . Some 
difficulty in reading was reported for 88% of the clubs, with the 
club average being 12% of members experiencing this problem. A 
certain amount of reading di f f iculty can be expected for any 
^roup, but considering the wide diversity of ages represented in 
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the sample, these figures were not unv xpected. The presence of 
the administrators was clearly needed to resolve as many of these 
reading problems as possible. 

Confusion about question format was slightly more prevalent. 
84.% of clubs experienced at least some difficulties here, with an 
average of 14% of the youth in a club being affected. This was 
primarily due to two sections of the Questionnaire, each 
requiring a branching of responses: youth who had any lifetime 
experience with cigarettes fand, separately, smokeless tobacco) 
were directed to one branch, while youth who had no experience 
were directed to another. Not all survey Questionnaires contain 
this level of complexity, but comprehensibi 1 i ty can be a problem 
for those which do. 

Another problem arose in the analysis of individual forms. 
A substantial proportion (about 6.1%) needed to be discarded due 
to incomplete survey forms or suspect patterns of responding. 
Responses to att itude items by the your ^er members were . in some 
cases, relatively flat (all "l"s, or all "5"s, etc.). This led 
us to infer that toward the end of the Questionnaire, these 
youngsters were either t iring, experiencing some processing 
difficulty, or not taking the Questions seriously. 
2. The one-day training for survey administrators pr-oved to be 
satisfactory . One area of concern before the survey involved how 
well the Advisors would be able to implement the survey after 

a short (one-day) training session, and how the 4-H leaders would 
respond to requirements for club Preparation and scheduling. 



14 



However, the club leaders were universally rated as supportive of 
the activity, and as completing all or most of the necessary 
preliminary steps correctly. 30% of the survey administrations 
were rated as "very smooth", 55% were rated as "smooth"* were 
rated as ''difficult" and 1% as "very difficult". One of the 
reasons for this successful outcome is undoubtedly that we 
explicitly recruited Advisors who were interested in incr^asina 
their skills and experience in research '=^ctivit ies, 

3. Ensur^ing pr^ivacv proved to be di '^''f' i ru 1 1 in tHf> rlub ^e^-ting . 
The informal, social atmosphere of the club meeting proved 
difficult to suppress for the survey administration activity. 
Talking among respondents was a moderate Problem. At many sites, 
survey administrators expended considerable effort to control 
this tendency. 

Distractions proved to be a problem at cerx;*9in club sites. 
Most of the distractions stemmed either from the adults present 
or simultaneous activities for children under age 9 who were not 
taking the survey. (In the majority of sites, these activities 
either did not go on or were conducted in a separate room in the 
facility.) Furthermore, despite Planned activities for early 
finishers specifically intended to maintain cjuiet conditions, the 
level of distracting noise tended to be greatest near the end of 
the administrat ion period as more members completed their forms . 
All tol d, the ciuiet of a classroom test -taking environment Proved 
to be a somewhat unrealistic expectation in this informal, non- 
classroom setting. ThJ 5 finding was one factor in our later 
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decision to change data collection formats for later study 
Phases. 

4. Absenteeism uias a problem in the survey administr^at ion . 
While schools do not experience perfect attendance rates, the 
compulsory attendance at school makes it a standard that is hard 
to match elsewhere. d-H is a voluntary orgafiizat ion, and members 
and their families are much more likely to skip a meeting if 
their schedules take them elsewhere. This was a concern of ours 
prior to the survey, and for the rr.ost Part it was borne out. The 
attendance rates at the meetings at which the survey was 
administered ranged from dO% to 100%, with an average of 63%. 
Make-UPS are particularly difficult in this setting, since 
meetings are infrequent and sites are geographically separated. 
The participation rate obtained in the survey compared favorably 
with many mail Questionnaires, but, nevertheless, was not 
considered totally satisfactory for our research needs. 
Decisions relating to future project phases 

The in-group adminstrat ion option was implemented mainly 
because we had no other choices for that phase in the project: 
the telephone approach was too costly and the mail approach was 
too problematic with young children in terms of 

comPrehensibili t y, timelines, and expected response rates. In 
addition, we wanted to explore the feasibility of the in-group 
approach for later program phases. Most of the data quality 
difficulties were resolved through a stringent post-screening of 
the Questionnaires. 

16 
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However, for the next phase of the project a different set 
of circumstances was present. The youth would be exposed to an 
educational Program, and tracked over several years, using at 
least 2 datapoints. Thus, our "research investment" in each 
child would be higher, and a 20-30% attrition at each measurement 
period would not be experimentally tolerable. Based on what we 
learned from the 1986 survey, we decided to collect data via 
telephone interviews* and contracted with an outside telephone 
survey center to conduct the calling* As an example of the high 
costs involved, based on our condition of a computer-assisted, 
20-minute interview, the agencies we contacted Quoted charges 
ranging from $30 - $50 per completed interview. 
Conclusion 

This paper has presented an analysis of three formats for 
collecting survey data from youth — mail, telephone, and in- 
group — as well as our own project experience regarding an in- 
group survey administration. The decision among the three must 
balance the need tor rigor with the pragmatics of budget, staff, 
and available time. When the in-group approach was implemented 
in an out -of -school setting, several problems arose which 
increased the need for strict post-processing. The telephone 
interview is judged to be the best option in terms of ensuring 
high participation and maximizing comprehensibi 1 it y of items. 
Thus it may yield the highest Quality data, but it is cost ly . 
Completion of the next Phase of our project wi 1 1 al low for 
another dimension to be added to this analysis. 
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Table 1 



Comparison of Three Data Collection Foraats 
on Selected Dieensions 



Foraat Cost 
In-group Low 



Tiae 

Required 

One day, 
plus sakeups 



Training Data 
Collectors 

Requires 

skilled 

administrators 



Comprehensibility 

Moderate : reading 
skill a concern 
but support ccn 
be available 



Participation 
Rates 

Moderate: 
Absenteeism 
can be a 
problem 



Denand 

Characteristics 

Potentially 
problematic 



Telephone Very Extensive, Requires Good: no reading Good, if Unt«ted 

high makeups must skilled required enough 

be scheduled interviewers callbacks 

are allowed 



Mail 



Low to 
moderate 



Extensive, 
three or more 
mailings must 
be planned 



No data 

collectors 

required 



Poor: no support 
for poor readers 
is a serious 
problem 



Poor: 

obtaining 

satisfactory 

r)articipation 

is the major 

problem 



Good: format 
allows for 
privacy in 
responding 
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